Learning Nearest-Neighbor Quantizers from Labeled Data by Information Loss Minimization

This paper proposes a technique for jointly quantizing continuous features and the posterior distributions of their class labels based on minimizing empirical information loss, such that the index K of the quantizer region to which a given feature X is assigned approximates a sufficient statistic for its class label Y . We derive an alternating minimization procedure for learning a finite set of prototypes that partition the feature space via the nearest-neighbor encoding rule with respect to the Euclidean distance, while simultaneously partitioning the simplex of class distributions by the nearest-neighbor rule with respect to the Kullback-Leibler divergence. The resulting quantizer can be used to simultaneously encode unlabeled points outside the training set and to predict their posterior class distributions, and has an elegant interpretation in terms of universal lossless coding. The promise of our method is demonstrated for the application of learning discriminative visual vocabularies for bag-of-features image classification.